DOCUMENT RESUME 



ED 452 589 



EA 030 962 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



AUTHOR 

TITLE 



PUB DATE 
NOTE 



IDENTIFIERS 



Ecton, Gayle W. ; O'Phelan, Mary H. ; Norman, Antony D. 

Three Years of Graduation 2010: Evolution of an Evaluation 
Plan . 

2000 - 11-00 

13p.; Paper presented at the Annual Meeting of the Mid-South 
Educational Research Association (29th, Bowling Green, KY, 
November 15-17, 2000). 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 

MFOl/PCOl Plus Postage. 

* Academic Achievement; *Curriculum Development; Educational 
Innovation; Educational Practices; Elementary Secondary 
Education; Evaluation Methods; * Evaluation Problems; 

Outcomes of Education; Program Evaluation; *Program 
Implementation 

Daviess County School System KY; Kentucky 



ABSTRACT 



Drawing on recent research in brain development, the public 



schools district in Daviess County, Kentucky, has developed a comprehensive 
program with the purpose of increasing students ' capacity for learning and 
achievement. The program called Graduation 2010 was conceived by educators 
and members of the community during 1997. The eight components, or "strands," 
of the program are now separated into three categories . The category referred 
to as Enhancers includes four "strands" intended to enhance learning (music, 
the arts, foreign language, and thinking skills) . Another category is called 
Support and includes two "strands" designed to increase support for schools' 
efforts to increase student learning (family involvement and community 
involvement) . This paper briefly describes the background and components of 
Graduation 2010, and the original evaluation plan as formulated by the 
research team together with the project steering committee. Also discussed 
are some of the challenges and problems encountered with developing and 
conducting the evaluation plan adopted for future evaluation. Abbreviated 
results of evaluation efforts during the first 3 years are given, including 
the results of surveys to measure the degree of implementation of the program 
and the results of interviews conducted with teachers about the program. 

(TEJ) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



EH O30<?£2 



i 



OS 

00 

</-) 

(N 

W-) 



Q 



W 



Three Years of Graduation 2010: Evolution of an Evaluation Plan* 
Gayle W. Ecton, Mary H. O’Phelan, and Antony D. Norman 
Western Kentucky University 



U S DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□P^This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

Gr ■ EcJ-Q n 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




Presented at the November, 2000 Mid-South Educational Research Association Annual Meeting, Bowlrng Green, KY. This is 
working copy. Please do not cite without permission. Comments, questions, or su^eshons should be addressed tc > Gayle W. 
icon. Department of Educational Leadership, Western Kentucky University, One Big Red Way, Bowlrng Green, KY 42104. 
'elephone (270) 745-4999 or (270) 745-4890. Email: gayle.gct0n(g|wkn t cdu 

o BEST COPY AVAILABLE 



2 



Abstract 

Drawing on recent research in brain development, the public school district in Daviess 
County, Kentucky has developed a comprehensive program with the purpose of increasing 
students’ capacity for learning and achievement. The program called Graduation 2010 was 
conceived by educators and members of the community during 1 997 . The eight components or 
‘strands’ of the program are now separated into three categories. The category referred to as 
Enhancers includes four ‘strands’ intended to enhance learning (music, the arts, foreign 
language, and thinking skills). Another category identified as Remove Barriers includes two 
‘strands’ intended to reduce barriers to student learning (reading/language development and 
health/emotional health). The final category is called Support and includes two strands 
designed to increase support for schools’ efforts to increase student learning (family involvement 
and community involvement). This paper briefly describes the background and components of 
Graduation 2010, and the original evaluation plan as formulated by the research team together 
with the project steering committee. Also discussed are some of the challenges and problems 
encountered with developing and conducting the evaluation plan during the first three years of 
the project, and the resulting modified evaluation plan adopted for future evaluation. 

Abbreviated results of evaluation efforts during the first three years are given, including the 
results of surveys to measure the degree of implementation of the program and the results of 
interviews conducted with teachers about the program. 
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Evaluation of Graduation 2010 

In this paper, we will present the background and description of the major components of 
Graduation 2010, the research plan as it was initially conceived and how it has evolved, and 
some of the key findings from evaluation of the first three years of the project. We will also 
discuss some of the issues inherent in attempting to reach conclusions about a program that is 
continuously transforming and how that has impacted the research plan design. 

Background and Essential Components 

The Graduation 2010 program is a curricular program developed by the Daviess County 
Public School system located in Daviess County, Kentucky. In 1997, a steering committee of 
educators, parents, and community members researched relevant information on brain 
development to determine what curricular changes could be made that would enhance 
intellectual capacity and remove barriers to learning of children in the school system. Naming its 
effort Graduation 2010 for the graduation year of the children who would enter kindergarten in 
the fall of 1997, the committee identified eight areas where educational changes might prove 
beneficial. These eight areas are also referred to as strands. In the past year, the school district 
has twice changed the categorization of these strands. There are now three categories: 
‘Enhancers’, ‘Remove Barriers’, and ‘Support’. The ‘enhancers’, which provide enhancements 
to the curriculum to increase the learning capacity of students include music, the arts, foreign 
language (now referred to as ‘second language’), and thinking skills. The strands identified as 
‘remove barriers’ seek to remove the barriers to student learning and include reading and 
health/emotional health. The third category referred to as ‘support’ is designed to increase 
support for schools’ efforts to increase student learning and includes the strands of family 
involvement and community involvement. 

Eight committees were formed to further study each of the eight areas and to develop 
implementation goals and guidelines. A review of the empirical evidence used by each 
committee is presented in the first year implementation report (Norman, O’Phelan, & Ecton, 
1998). 

The eight committees submitted short-range and long-range goals for Graduation 2010. 
What follows is an identification of the short-range goals. For Music, goals included allocating 
space, training teachers, and establishing keyboard labs in each school. For the Arts, goals 
included identifying an arts facilitator for each school and coordinating at least nine arts 
experiences for kindergarten. For Foreign Language (Spanish), goals included training teachers 
in Spanish, developing instructional videotapes, and beginning instruction in basic Spanish 
vocabulary. For Thinking Skills, teachers were trained in Talents Unlimited (Chissom & 
McLean, 1993) and were to begin implementing strategic games such as chess, into the 
curriculum. For Family Involvement (Parental Involvement), several strategies were 
recommended including new student orientations and homework committees composed of 
teachers and parents. For Health/Emotional Health, goals included providing each school a 
nurse one day a week and developing fitness programs at each school. For Reading/Language 
Development (Reading), the goal was to implement the Kentucky Reads program which targets 
primary children who are at least one grade level behind in reading. For Community 
Involvement, the goal was to secure a corporate sponsor from the community for the 2010 
kindergarten class. 
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A final goal of the steering committee was to obtain a research team that would study 
both the level of implementation of the goals at each school and the overall effect of this 
program on student intellectual development and achievement. The committee contacted the 
Dean of Western Kentucky University’s (WKU) College of Education and Behavioral Sciences 
and requested assistance. The research team was created when the Dean asked us to consider 
working with the school district’s steering committee, superintendent, and administrators to 
develop a research evaluation plan. As a signal of its commitment to the evaluation process, the 
Daviess County Public Schools hired a research and assessment coordinator for the school 
district who would also serve as a liaison between the WKU research team and the school 
district. 




Developing a Research Plan 

During the first year of Graduation 2010, the research team met several times with 
administrators and steering committee members in Daviess County to plan an evaluation strategy 
that would be responsive to the needs of the program. Most of these meetings were dedicated to 
clarifying the goals and procedures associated with the project. We faced some early difficulties 
because we were not involved in the initial planning of the program and entered the process after 
implementation had already begun. As a result, it took nearly a full year of communication 
before we could come to agreement about what we would be looking for, what information we 
would need to gather, and how to go about gathering it. (The late start during the first year of 
implementation limited the initial evaluation effort to a survey to gauge the extent to which the 
intended steps of the program had been implemented.) 

The complexity of evaluating this project quickly became apparent. Each strand of 
Graduation 2010 had at least one recommendation and each recommendation had a list of 
implementation steps. The list of implementation steps ranged in number from a minimum of 3 
in the Arts strand to a maximum of 29 in the Family Involvement strand, and in some cases, 
implementation steps were written such that they included various parts. The total 
implementation steps was 107. To further complicate the issue, implementation steps were not 
necessarily equal in weight, such that the implementation of only one step might be more 
significant than implementation of several steps. Furthermore, the scope of the program requires 
much cooperation and planning, and it was never intended to be implemented all at once. Also, 
implementation was not required of any school, and decisions about what and when to 
implement were left up to the individual school. 

During the first year and continuing into the second year, there was much discussion 
about what the elements of the project included, what they were trying to accomplish, and what 
indicators of success could be used to evaluate the project. Originally, committee members 
wanted student scores as the bottom line, but gradually, we started hearing ‘capacity to learn,’ 
which we interpreted to mean aptitude, as well as achievement. We started with the following 
indicators for achievement: Woodcock- Johnson, Talents Unlimited tests, SAT scores, 
Kentucky’s state assessment scores, CTBS/5(TerraNova), ACT scores, Advanced Placement 
tests, and Scores on the Primary Test of Cognitive Skills (P/TCS). The school district already 
uses most of these tests, and they agreed to provide the scores to us as they became available. 

These discussions resulted in 1 0 research questions, each of which had several different 
indicators. A list of indicators by research question is presented in Table 1. We decided on a 
variety of information to help us assess the overall impact of the program. We wanted to do 
more than count the things that are easily counted (e.g., how many minutes per week of music 
instruction). We recognized that many other things could impact the degree and quality of 
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Table 1. Research Plan for Graduation 2010 
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Research Question 


Instrumentation 


Time Frame 


1 . To what degree has each strand of the 
program be implemented? 


Survey of principals 
School visitations 

# of teachers trained in Talents Unlimited 
School data sheets 


Yearly 


2. To what degree has the learning of 
children been increased? 


P/T.S. 

Woodcock Johnson (individuals) 
Talents Unlimited Tests 


Yearly 


3. Has the program increased achievement? 


KIRIS (CATS) 

CABS/5 (Terr^Nova) 

# of students in upper level courses 
Advanced Placement tests 

# of students taking SAT/ACT/SAGE 
SAT/ACT/SAGE tests 

Kentucky Reads results 
% of students reading at or above grade level 
School data sheets 


Yearly 


4. Has the school learning environment 
improved? 


School culture audits 
Student/teacher/parent interviews/surveys 
Attitude surveys of music, arts, foreign language 


Yearly 


5. Has family involvement increased? 


Parent/student/teacher surveys 
School data sheets 

# of PTO members 

# attending school meetings 

# of parent volunteers 


Yearly 


6. Has the health of students improved? 


# of absences 

Demographics (pregnancies, etc.) 
Parent/teacher/student surveys 


Yearly 


7. To what extent are students well educated? 


School data sheets 

# taking foreign language, AP classes, art, music, 
upper level courses 

# and % taking ACT/SAT 
ACT/SAT scores 

# and % of dropouts 

U and % in college prep track 

# and % going to college 

# receiving scholarships 


Yearly 


8. Are students benefitting from the arts? 


CATS performance assessment 
School data sheets 

# taking private music lessons 

# enrolled in fine arts and music class 

# in arts clubs and activities 


Yearly 


9. Are students benefitting from foreign 
language? 


School data sheets 

# enrolled in foreign language classes 

# participating in foreign language clubs and activities 


Yearly 


10. Has community involvement increased? 


Teacher/student/administrator surveys 
Letters from sponsors 
Documentation of sponsor participation 


Yearly 



6 BEST COPY AVAILABLE 



6 



ERjt 



implementation of the program. We knew we needed some qualitative evaluation to find out 
how different stakeholders feel about the project, whether or not teachers, students, and parents 
are buying into it, and even general atmosphere and level of cooperation and collegiality at 
individual schools. We agreed that data analyses would have to include qualitative descriptions 
of survey and interview responses, quantitative comparisons of test scores and frequency 
variables, and trend analyses for tested groups and individuals. Results would be reported at 
yearly intervals for the program as a whole, by school, and by program strand. 

We asked for information such as number of students taking the SAT, enrolled in 
Advanced Placement (AP) courses and taking AP tests, or enrolled in higher level courses. We 
asked for the number of students going to college or getting scholarships, dropping out of school, 
the number of pregnancies during high school, the number of students taking music lessons 
outside of school, choosing to study abroad, or getting involved in foreign language, art, or 
music clubs. We included surveys and interviews, as well as data sheets (containing some of the 
information listed above) to be completed by the schools, school culture audits, and individual 
testing for a sample of students. 

There were some limitations on the research design as well as the data gathering 
procedures worth noting. First, the absence of any allocation of funds for the evaluation of the 
program affected decisions about data gathering and time for the research team. Second, the 
district administrators did not want to overburden faculty and staff with data collection 
procedures by asking them to do more paperwork than they already have to do. Third, 
administrators felt it would be inappropriate for the research design to include a control group, 
either from within the school system or from a comparable district elsewhere. Finally, they did 
not want surveys to go out to community at large but only to staff and students of the county 
public school system since a large independent school system operates within the same county, 
contributing to a politically sensitive environment. 

Evaluation Begins 

As previously mentioned, during the first year the evaluation focus was on 
implementation (see Norman, O’Phelan, & Ecton, 1998). Principals were given an open-ended 
survey halfway through the first year on which they were asked to report progress in each of the 
eight strands of the program. At the end of the first year, principals were asked to rate to what 
extent each of the goals for that year had been accomplished in their schools. A similar survey 
was given at the end of year two with modifications made to accommodate the dynamic aspect of 
the program (see O’Phelan, Norman, & Ecton, 1999). Likewise, the survey used in the third year 
underwent some modifications to better reflect the changes which continued to be made to the 
program (see O’Phelan, Norman, & Ecton, 2000). The superintendent was also given a survey 
each year to rate overall implementation progress in each of the eight strands. 

During year two, we began collecting baseline data to measure student intellectual 
change as the program progresses. We individually tested those students who were part of a 
sample of 150 kindergarten students for whom the schools had received parental permission. We 
were able to test 1 15 out of the original 150 students in the sample. An additional sample of 
kindergarten students were tested during year three. 

Also, during the latter part of year two, we conducted school culture audits for all 12 
elementary schools. When the results of the culture audits were presented to each school s 
faculty, researchers interviewed the teachers about positive and negative changes occurring in 
the school, problems with teaching and learning, suggestions for improvement, and finally, then- 
opinions about Graduation 2010. 
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To gather data required from individual schools, we devised a data sheet to be completed 
by the schools during the school year. So as not to overburden school personnel, the district 
administration requested that we consolidate this data form to include all of the information we 
could not otherwise obtain. Schools were to progressively complete the form during the year as 
data became available and return the completed form to us at the end of the year. 

First Year Implementation 

Mid-year and at the end of the first year, principals reported their progress toward 1997- 
98 goals by rating each goal as not implemented at all, partially implemented, or fully 
implemented. Not surprising, considering the voluntary nature of the program, progress of 
schools in implementing the 1997-98 goals at mid-year reflected an uneven pattern. Two strands 
(Music and Health/Emotional Health) were reported to be fully implemented, although 
information about Health/Emotional Health was missing from one school. Seven schools 
reported full implementation of the Kentucky Reads program. Some progress was reported on 
the remaining strands, with between four and ten schools reporting some level of activity. Eight 
of the 12 schools reported some progress toward the Arts goals, one reported full implementation 
and three reported no implementation. There was a similar response on the Thinking Skills 
component. Nine schools reported some level of progress on the goals for Foreign Language and 
three reported none. For Family Involvement, 10 schools reported some implementation and two 
reported no progress toward the goals. Although there was no specific school level goal for 
Community Involvement since the general goal was to obtain corporate sponsorship for each 
class at the district level, in response to a question about Community Involvement, 10 schools 
reported some activity, one reported none, and information from one school was missing. 

At the end of the year, 1 1 of the 12 schools reported using the same survey format. Ten 
of the 1 1 schools reported full implementation of the three goals for the Arts strand (identify an 
arts facilitator, provide nine arts experiences, and provide a Riverpark experience). All 1 1 
schools reported full implementation of the Music goal (install keyboard labs in the school), and 
10 reported having a school nurse at least once a week, a goal of the Health/Emotional Health 
strand. Between the midyear and end of the year reporting, there was a shift in the number of 
schools reporting full implementation rather than partial implementation on goals associated with 
Family Involvement, Foreign Language, Reading/Language Development, and Thinking Skills. 

When overall ratings were limited to the list of 1 997-98 goals, Music, Arts, and 
Reading/Language Development merited high implementation ratings, and Health/Emotional 
Health, Family Involvement, and Thinking Skills had medium ratings. Progress toward 
implementation was further evidenced by the shifts from low and medium categories at mid-year 
to medium and high categories at the end of the year. The complete results of first year 
implementation may be found in the report on Graduation 2010 (Norman, O Phelan, & Ecton, 
1998). 
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Second Year Implementation 

Principal surveys were also used to measure implementation in the schools by the end of 
the second year. The dynamic aspect of the project made it impossible to use the same survey 
items for year two as were used for the first year. Some of the steps required one time only 
activities (e.g., establish guidelines, hire personnel, allocate space, buy keyboards for music), and 
once completed, those items were no longer applicable. Some steps not yet introduced during 
the first year had to be added to the second year survey because at least some schools were 
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beginning to implement them. At the same time, some of the implementation steps on the 
original list had been eliminated as not feasible or desirable. Finally, some altogether new ones 
were added during the second year. 

Using the 1 997-98 goals as a standard, two strands could be considered fully 
implemented. All 12 schools reported full implementation of the original goals for Foreign 
Language and Music strands by the end of the second year. As many as 10 or 1 1 schools 
reported full implementation of some of the goals in the Arts, Family Involvement, 
Reading/Language Development, and Thinking Skills. All schools reported at least medium 
levels of implementation with respect to the 1 997-98 goals. 

However, when considering all items on the 1999 survey and looking at the overall self 
report data for the 12 schools reporting, two strands (Thinking Skills and Health/Emotional 
Health) appear to have been relatively low in implementation. For these two strands, there were 
no items for which at least 1 1 of the 12 schools reported full implementation and most items had 
less than half the schools reporting full implementation. Using a similar standard, Music would 
be considered moderate in implementation since only two of the 21 items had 11 or 12 schools 
reporting full implementation but 10 items had at least half of the schools in full implementation. 
Family Involvement could be said to have moderate implementation overall since three of the 22 
items were reported to be fully implemented by at least 1 1 schools, and very few items were 
rated at the lower end of the rating scale. The Foreign Language strand was reported as 
moderately high in implementation. At least 1 1 principals reported full implementation for two 
of the three Foreign Language goals. Finally, the strands for the Arts and Reading/Language 
Development were reported as high in implementation. Between nine and 1 1 schools reported 
full implementation on five of the six goals for the Arts, with very few ratings at the lower end of 
the scale. 

Analyses of these data were complicated by a number of factors. Since the strands had 
unequal number of steps (and survey items), it was somewhat arbitrary to classify each strand as 
low, moderate or high in implementation. The complexity of the program made it difficult to 
evaluate the implementation of individual strands. The program has been dynamic in nature and 
was intended to be implemented gradually. Implementation by each school has been voluntary 
and decisions about what and when to implement were left up to the individual school. 

Likewise, principals were not given definitions of different levels of implementation so what one 
principal considered ‘full implementation’ might have been considered only partial 
implementation by another principal. The complete results of second year implementation can 
be found in O’Phelan, Norman, & Ecton (1999). 

A Qualitative Analysis of Teachers’ Reactions 
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A concern regarding the implementation of this program was that the overall school 
climate might influence its effectiveness at each school. In the Spring of 1999, we conducted 
school culture audits for all 12 elementary schools. The audit results revealed that all schools 
were found to be healthy with very few discrepancies between the importance and presence of 
key elements of a healthy school climate. While meeting with each school to present the results 
of the culture audit, the research team led the school faculty through a discussion of nine open- 
ended questions and one rating question related to Graduation 2010. The questions generally 
dealt with what changes and problems teachers saw in their school as well as their opinions about 
Graduation 2010. 

Teachers cited a variety of problems and mentioned both positive and negative changes. 
Most of these had to do with (what was then referred to as) barriers to learning such as parenting, 
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health/emotional health, and student behavior problems. Teaching problems cited were more 
related to time constraints and instructional issues. On the other hand, discussion of changes in 
the schools and the most important aspects of services for students were generally positive and 
all enhancer strands of Graduation 2010 were mentioned. Three of the four enhancer strands 
(Arts, Spanish, and Thinking Skills) were mentioned as the best thing the school does. 

Finally, teachers were asked to rate on a scale of 0 to 10 (0 not important at all, 10 very 
important), the importance of Graduation 2010 to their school. Eight schools rated Graduation 
2010 as 8 or higher in importance. One school rated the program as 7 if Spanish and chess were 
included and 5 if they were not. One school rated Graduation 2010 between 2 and 5 in 
importance. 

Specific problems identified with Graduation 2010 were about time constraints, teacher 
training, and three of the four enhancer strands, especially Spanish. Music was not mentioned as 
a problem. A more detailed discussion of the results of this qualitative analysis is contained in 
the report by Norman, O’Phelan, & Ecton (1999). 



Third Year Implementation 

Surveys, similar to those used in the second year but again modified to accommodate 
changes in the program, were distributed to the principals in the 12 elementary schools and the 
superintendent. These administrators were asked to rate the implementation of the original 
goals of the project from no implementation to full implementation. The survey also asked for 
ratings on implementation steps that were added to the survey as a result of the program changes 
in 1999 and 2000. Based on survey results, implementation was classified as low, moderate, or 
high for each strand. 

When all survey items are included, two strands, Health/Emotional Health and Thinking 
Skills, were classified as low in implementation as rated by principals. Also, based on ratings of 
principals, Music, Reading/Language Development, Foreign Language, and Family Involvement 
were classified as moderate, and the Arts strand was classified as high in implementation. (The 
strand of Community Involvement was not included in the survey since the only goal was 
essentially to obtain sponsors for each succeeding Kindergarten class and this was a 
responsibility of the district.) The superintendent’s rating for overall implementation disagreed 
with the principals’ ratings in six of the seven strands rated, although the difference was 
considerable in only two instances. The superintendent’s rating was slightly higher for 
implementation in Family Involvement, Thinking Skills, and Reading/Language Development. 
His rating was slightly lower in the Arts. His rating was considerably higher in Music and 
Health/Emotional Health. The superintendent’s rating agreed with the principals’ rating in 
Foreign Language. Subjective interpretation of goal statements or survey items might also 
explain the fact that the superintendent’s general rating of implementation for each strand did not 
exactly agree with the classification of ratings derived from the survey of principals. 

However, if the measurement of implementation is limited to the original 1997-98 goals, 
strands high in implementation would be the Arts, Music, and Foreign Language. Family 
Involvement, Health/Emotional Health, and Thinking Skills would be considered moderate, and 
implementation in Reading/Language Development would be considered low. 

When Year Three implementation is compared with the original goals, the number of 
schools at each successive stage of implementation has increased. With the exception of 
Family Involvement goals, there were no schools reporting ‘no implementation’ at the end of the 
third year for any goal. On the other end of the implementation spectrum, however, even at the 
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end of the third year, most of the strands were not yet fully implemented in terms of the 1997-98 
goals for implementation. 

Analysis of these data continue to be complicated by the same factors which were 
discussed in the section above concerning Year Two Implementation except the problems 
become compounded with each succeeding year due to continuing changes in the program. 
Again, some items on the previous survey have been eliminated and new items added. Also, the 
goals for the project have not been revised since 1997-98. This has caused imperfect alignment 
between the survey questions and the list of goals to be evaluated. As a result, two different 
standards of comparison were used in the third year implementation report, the original 1997-98 
goals, and all 2000 survey items. It should be noted, however, that even though the list of 1997- 
98 goals has not changed, the list may no longer be a good measure of the degree of 
implementation since the emphasis of the project may have changed considerably over that time. 

When percentage of foil implementation of the 1997-98 goals is used as the standard, 
implementation at the end of year three in the Arts strand would be considered complete (100%); 
implementation in Music and Foreign Language would be considered high (81% and 83% 
respectively); implementation in Family Involvement (65%), Thinking Skills (58%), and 
Health/Emotional Health (58%) would be considered moderate, and implementation in 
Reading/Language Development (42%) would be considered low. It should be taken into 
consideration, however, that the percentages are based on few goals (and in some cases only one 
goal) and do not accurately represent what has been accomplished in any of the strands overall. 
The complete results and a more thorough discussion of third year implementation may be found 
in O’Phelan, Norman, & Ecton (2000). 



Problems Encountered 
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At the end of the third year of our participation in the project, a number of difficulties and 
problems had become evident. These included difficulty in obtaining some of the data needed, 
due in part to the voluntary nature of the program and to the extra time and effort required. The 
school district was not interested in some of the original research questions and preferred a more 
narrow focus, and thus, questioned the time necessary to gather some of the data requested. 

The voluntary aspect of the program also resulted in uneven levels of implementation of 
the program strands in different schools. What is considered foil implementation of the program 
varies from school to school. Even taking the variations into account during evaluation, it 
becomes more difficult to attribute successes or failures to any particular set of procedures. 

The school district treats this program as a transformational effort, changing and adapting 
the program as the demands of the environment require. While this is viewed as positive and 
necessary by the school district, aspects of the program keep changing so fast that it becomes 
difficult to maintain anchors upon which to base the evaluation of results. New goals and policies 
have been added and some original ideas abandoned as unsuitable or impossible. The constant 
change that is viewed by the school district as a continuous improvement effort severely affects 
the researchers ability to gather long-term data and to isolate what seems to be effective and what 
does not. 

The research effort could be more effective and beneficial to the school district if the data 
collection instrument accurately reflected current goals and practices in the district. This could 
be accomplished by updating goals annually and by modifying the survey accordingly. The 
research team believes that those participating in program implementation would benefit from 
discussions and review sessions in which the program goals of current activities are clarified or 
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modified to fit long term purposes of the district. This could also serve to renew the commitment 
of those actually implementing the program. 

Funding to support the research effort continues to be a problem, thus limiting the 
research team’s time and also affecting the scope of the research effort. Communication 
problems further complicate research efforts as changes have been made and the research team 
has not always been informed until well after the changes. All of these issues and other related 
issues are discussed more in depth in a report on performance indicators and problems 
(O’Phelan, Norman, & Ecton, 1999). 

Revising the Research Plan 

Acknowledgement of these problems by both the school district and the research team 
has led to numerous discussions which focused on how to revise the evaluation plan so that it 
- would provide the information needed for the researchers to answer what everyone agreed to be 
the essential questions. As a result of these discussions, the scope of the research plan will be 
limited to focusing on the ‘enhancer’ strands, fewer schools, and with emphasis on one school 
determined to be high in implementation of the total program. The school district will 
insure full cooperation of the schools evaluated and to furnish all data requested. 

Communication has already improved and efforts are being made to insure effective 
communication during all phases of this research initiative. 

The school district and the research team have refocused on the central issue of how to 
best measure the degree to which Graduation 2010 affected student achievement. Two central 
research questions have been articulated to be answered: 1. To what extent are the 2010 
initiatives being implemented in the Daviess County Schools?; and, 2. To what extent has 
student achievement increased? 

Data to answer these questions will be gathered through three methods. First, to answer 
question one, each year a survey will be developed based on the yearly goals developed by the 
Graduation 2010 steering committee. For each goal, school principals and the superintendent 
will be asked to rate the level of implementation as a school or as a district. Second, to answer 
question two, achievement data will be collected from schools to longitudinally measure the 
progress of schools and individual students. Achievement data will include school and 
individual state assessment (CATS) scores, CTBS/5 (TerraNova) scores, Talents Unlimited test 
scores, Primary Test of Cognitive Skills (P/TCS) scores, and Woodcock-Johnson scores. Third, 
to develop a richer understanding of the trials and triumphs in implementing this program, the 
research team will take a closer look at one school with a diverse student population and 
recognized as high in program implementation. Using a qualitative case study approach, the 
researchers will devote up to eight days per year observing and interviewing the school’s faculty, 
staff, administration, students, and parents. Data analyses will include quantitative and 
qualitative descriptions of survey and interview responses, quantitative comparisons of test 
scores and frequency variables, qualitative observation reports, and trend analyses for tested 
groups and individuals. Results will be reported at yearly intervals. Assessment is intended to 
be both formative and summative. Results will be shared with stakeholders at all levels so that 
thoughtful modifications can be made to increase the likelihood of reaching Graduation 2010’s 
intended outcomes (Ecton, O’Phelan, & Norman, 2000). 
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